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Abstract Body 



B ackground/context : 

Despite considerable evidence regarding the importance of pedagogical content knowledge, little 
is known about the conditions that foster its growth (Baxter & Lederman, 1999; Loughran, 
Mulhall, & Berry, 2003; Magnusson, Krajcik, & Borko, 1999; Van Driel, Verloop, & De Vos, 
1998). Although some teachers acquire pedagogical content knowledge through years of 
experience (Geddis, 1993; Gudmundsdottir, 1987a, b; Shulman, 1986), this process is time 
consuming, and not all teachers benefit from experience in the same way. What is needed are 
principles to guide the design of teacher courses that reliably develop pedagogical content 
knowledge in an efficient and timely manner. 

Research has yielded consistent evidence that well-designed student curricula, combined with 
intensive professional development, can result in changed classroom practice and student 
learning (e.g., Cohen & Hill, 2001; Fennema et al., 1996; Franke et al., 2001; Roberts, 1996; 
Saxe, Gearhart & Nasir, 2001; Wilson & Sloane, 2000). Such studies have found that an 
important factor is the degree to which subject content is explored in depth in the professional 
development and is embedded in student curriculum. However, research has made far less 
headway in explaining how it is that particular kinds of professional development activities yield 
results — or not — at the classroom level. Questions thus remain largely unanswered about how to 
structure teacher education experiences that foster the development of pedagogical content 
knowledge. In their conclusion of a comprehensive review of research on teaching, Bransford, 
Brown, & Cocking (2000) recommend that teacher education courses be developed to provide 
teachers with more effective discipline- specific pedagogical knowledge, and that research 
address important questions about how to do this. These questions regarding how to combine 
content and pedagogy to produce the greatest impact on teaching and learning represent the 
central problem that motivated the research reported here. 

Purpose/objective/research question/focus of study: 

Few elementary school teachers command a science background sufficient to offer children early 
and sustained opportunities to develop an understanding of science concepts, and facility with 
scientific modes of reasoning. This four-year, NSF-funded study was designed to investigate 
how and to what extent particular combinations of science content and pedagogical learning 
experiences for teachers in grades 4—5 produce an impact on teachers’ knowledge for teaching 
and on their students’ learning. In this paper, we report results from the first phase of this work, 
which addressed the fundamental questions of whether the professional development courses did, 
in fact, lead to teacher and student knowledge gains, and what the effects were for the three 
course models that were tested, in relation to the control group and to one another. 

Setting: 

This experiment was conducted at eight national research sites, four in the western United States 
(in Arizona, California, and Washington), and four in the eastern (in Massachusetts, North 
Carolina, and Alabama). Each research site was either a large school district (three sites) or a 
collection of geographically close districts (four sites with four or five districts each, and one 
with 34 districts), for a total of 54 districts in six states. Geographically close districts typically 
had a history of working together, with the support of a regional entity such as a regional subject 
area project, university, or county connection. 
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Potential research sites were identified through recommendations from colleagues in networks of 
science leaders throughout several states (e.g., lead teachers involved in local NSF- and state- 
funded Math-Science Partnership Projects). Recruitment criteria included: (a) well-established, 
stable district or regional science program, so participants would not be teaching science for the 
first time; (b) strong science leadership (e.g., staff developers, teacher leaders, and district 
staff/vision), from whom to draw local course facilitators, so as to test the courses’ effects when 
delivered by professionals in the field (not the course developers); (c) academically, culturally, 
and linguistically diverse student population; (d) standards-based curriculum for teaching science 
in place, along with necessary supporting resources for teachers and students (and variety in 
curricula across districts); (e) strong interest in and philosophical alignment with the professional 
development approaches to be tested; and (f) proven ability to recruit teachers for professional 
development. 

Population/Participants/subj ects : 

A sample of 268 fourth grade teachers originally met the selection criteria of teaching fourth 
grade electric circuits in both 2007-08 and 2008-09 school years and agreeing to provide study 
data, and eventually provided complete data sets including pre- and post-test data from over 
5,000 students. Students were not randomly assigned, instead were the students in all 
participating teachers’ classes. The teachers had been randomly assigned to the four treatment 
conditions, within site, within school, and received a $650 stipend per participant, plus an 
additional stipend if they participated in an intensive classroom sub-study or in the follow-up 
data collection. Table 1 shows the number of teachers randomly assigned to each group at each 
site, and the number who continued into a follow-up year. 

[Please insert Table 1 about here.] 

Because participants were randomly assigned to groups, teacher backgrounds were similar across 
the four conditions with respect to years teaching experience, years teaching circuits, hours of 
professional development in science, and hours of professional development in circuits. The 
sample included teachers with a wide range of teaching and professional development 
experience, spanning from novice to veteran. 

Intervention/program/practice: 

The study employed three experimental professional development course models, each of which 
was modeled after a different approach currently supported by districts and teacher educators. 
Each of the three course designs encompass eight three-hour sessions completed over a series of 
days — over 8 to 14 weeks during the school year, or a five-day period during the summer. The 
courses focused on the teaching of electric circuits, a common component of elementary school 
curricula. All three course designs contained core features of effective science professional 
development, based on evidence in a growing body of literature (Borko, 2004; Cohen & Hill, 
2001; Garet et al., 2001; Weiss et al., 1999). 

Each course consisted of two portions — a science content component and a set of strategies 
intended to develop pedagogical content knowledge. Every course model involved teachers in a 
common set of science investigations that enabled in-depth, collective exploration of science 
content. However, the models varied from one another with regard to additional activities 
designed to support the development of pedagogical content knowledge. In Course A, Teaching 
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Cases, teachers read and discussed written narratives containing student work and dialogue, 
teacher thinking, and descriptions of instructional materials and activities. Course B, Looking at 
Student Work, involved teachers in analyzing the affordances of specific classroom assessment 
tasks, and discussing evidence of students’ scientific thinking drawn from their own classrooms. 
Course C, Content Immersion, combined the science investigations with a meta-cognitive 
learning analysis of the teachers’ own science learning processes. Teachers in the Teaching 
Cases and Content Immersion courses taught electric circuits units following completion of the 
course; teachers in the Looking at Student Work course taught the unit concurrently with the 
professional development over a two-month period so as to supply samples of their own 
students’ work. 

Research Design: 

This was a cluster-randomized experimental design with repeated measures over a two-year 
period. The three professional development courses were delivered 8 times each during the 
study for a total of 24 times during the study, 12 in Round 1 and 12 in Round 2. Table 2 shows 
the counterbalanced research design we used to control for order effects and to allow analysis of 
both facilitator and treatment effects, without confounding the two. Table 3 shows the numbers 
of courses offered, teachers in each group, and approximate numbers of students from whom we 
collected content test data. 

Teachers at each site were randomly assigned to one of four treatment groups. After these 
assignments, a sub-sample of 36 teachers was randomly selected to participate in an intensive 
classroom study — 12 from each of three sites (3 per treatment group). These intensive study 
participants and half of the original sample of treatment teachers were then randomly selected to 
provide follow-up data one year later. (Teachers in the control group were not included in the 
follow-up study so they could receive the professional development course in summer 2008.) 

[Please insert Tables 2 and 3 about here.] 

Data Collection and Analysis: 

The data collection measures, samples, and procedures are summarized in Table 4. A teaching 
background survey provided data on all teachers’ professional experience and perspectives on 
science teaching. Tests of electric circuits content knowledge were administered to all teachers 
pre- and post-PD, and to their students before and after the electric circuits unit. A randomly 
selected sub-sample of teachers participated in pre- and post-PD interviews designed to elicit 
pedagogical content knowledge, and were observed and video-taped twice each while teaching 
lessons on electric circuits. All professional development activities were video-taped, and 
survey, content knowledge, and interview data were collected from the professional 
development facilitators. Data were collected in two rounds of professional development course 
implementation conducted from August-December 2007 and from January-June 2008. The 
resulting data set is large, rich, and complex. 

Findings / Results: 

Teacher outcomes. We fitted hierarchical linear models to analyze the impact of the teacher 
professional development courses on teacher and student content test scores and found that all 
three treatments caused large test score gains for both teachers and students. As shown in 
Table 5, teachers who took any of the courses achieved considerably higher post-test and gain 
scores than teachers in the control group. On average, teachers in all three treatment groups 
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showed gains of about 22 percentage points, whereas the control group’s gain was less than 3 
points. Furthermore, as shown in Table 6, the gains that were maintained an additional year after 
the professional development were still far greater than the control group gains in Year 1. 

[Please place Tables 5 & 6 about here] 

As shown in Table 7, hierarchical linear modeling (HLM) confirmed that all three treatments had 
a significant positive effect. We see here that control teachers’ scores increased about 1.6 
percentage points from pre- to post-test, and scores of teachers in all three courses increased 19- 
20 points on top of that. Effect sizes were quite high — 1.92, 1.91, and 2.03 for Teaching Cases, 
Looking at Student Work, and Content Immersion, respectively. (Effect size was computed as 
the coefficient divided by the pooled standard deviations of the teacher gains.) No differences 
were found among the three courses in their impact on teacher content knowledge. 

[Please place Table 7 about here] 

Student outcomes. As shown in Table 8, students of teachers in the three treatment groups 
demonstrated mean gains of approximately 20 percentage points from pre- to post-test on the test 
of electric circuits content knowledge, whereas students of control group teachers gained an 
average of 13 percentage points. Furthermore, at every research site, students of treatment 
teachers achieved mean gain scores that appear higher than the overall mean gain for control 
teachers’ students (see Table 9). HLM results (Table 10) again confirmed these results — effect 
sizes for impact of the treatments on student test scores were 0.39, 0.52, and 0.52 for Teaching 
Cases, Looking at Student Work, and Content Immersion, respectively. (Effect sizes were 
measured as a ratio of treatment effect and the pooled standard deviation of student gain. This 
latter value is a measure of how much variation in gain we get from students without controlling 
for any covariates. This is the most conservative measure of effect size.) Although effect sizes 
were higher for the Looking at Student Work and Content Immersion courses, tests for 
differences among the three courses were inconclusive. 

[Please place Tables 8, 9, and 10 about here] 

Accounting for Student Content Knowledge Outcomes. An important question is whether the 
courses’ impact on teacher content knowledge alone accounts for the variance in student 
outcomes. That is, is it sufficient to bolster teachers’ content knowledge in science as a means of 
producing student learning gains? To determine this, we compared another HLM model, which 
had only teacher content knowledge, to the model that had both teacher content knowledge and 
the treatment dummy variables. The models were indeed different (p < 0.01), and all three 
treatment effects are significantly positive ( p < 0.05, p < 0.005, p < 0.005 for Teaching Cases, 
Looking at Student Work, and Content Immersion, respectively). The model coefficients indicate 
expected additional student gains beyond those gains due to the teachers ’ content knowledge of 
3.4, 5.4, and 5.3 percentage points (effect sizes 0.26, 0.42, and 0.41, respectively). We conclude 
that all three teacher treatments do something to improve student test scores beyond that of 
merely improving teachers’ content knowledge. This additional factor accounts for at least as 
much of the final student impact as does the gained content knowledge of the teachers. 
Preliminary analyses of our other data indicate that the additional benefits constitute 
improvements in teachers’ pedagogical content knowledge.To further account for student 
knowledge outcomes, we have also analyzed teacher interviews and classroom observation data 
to begin identifying differences in pedagogical content knowledge and teaching practices of 
teachers in the treatment versus control groups, as well as among the three treatment groups. 
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Preliminary results indicate that teachers who took any of the three courses exceeded control 
teachers in their explicit attention to developing students’ conceptual understanding of science 
ideas. Treatment teachers more often identified learning goals in terms of conceptual 
understanding (e.g., stating that they want the students to be able to explain the difference 
between series and parallel circuits rather than simply naming topics in the unit). In their lessons, 
treatment teachers used more representations that make patterns in data visible for analysis by 
the students and support students’ scientific reasoning about the data. 

Conclusions: 

Using a randomized experimental design, we were able to establish that all three professional 
development models brought about large gains in teachers’ and students’ science content 
knowledge, well beyond those of comparable control groups, and that effects of the courses 
persisted a year later. Clearly, the courses have design features that are highly effective at 
preparing teachers to support their students’ science learning. We also found that it is possible to 
achieve effect sizes of 2.0 for teachers and .5 for students with high quality professional 
development experiences and a rigorous research design. 

An intriguing finding is that the three courses, which differ considerably in design features, did 
not differ significantly in their effects on teacher and student content learning. This result 
suggests that important elements in professional development can be configured in a number of 
ways and still have beneficial effects. The presence of certain characteristics is essential for high 
quality professional development, but it is possible for professional development to embody 
them in a variety of effective ways. 

The fact that the courses did not differ in impact on teacher or student content test scores does 
not preclude the possibility of differences in impact on pedagogical content knowledge — 
differences that could prove important for intentional design of professional development geared 
to be effective in varied learning contexts and for student populations with particular 
characteristics. Since the courses were so different with respect to their pedagogy-related 
components, in fact, it seems very likely that they affected teachers’ knowledge for teaching in 
different ways. Current analyses are examining these possibilities. 

Finally, we found that the large impact of the courses on teacher content knowledge alone does 
not explain so high a proportion of the variability in student performance as to suggest that 
teacher content knowledge alone is responsible for the student content gain. There are other 
factors that we are currently investigating. 

Because we found such large effects on teacher and student learning, subsequent analyses of the 
data set from this study will (a) examine relationships among gains in teacher content and 
pedagogical content knowledge, changes in classroom practice, and student learning; and 
(b) work toward specifying processes by which science professional development achieves its 
effects on teaching and learning. Pursuing these goals can enable us to make a strong empirical 
and conceptual contribution to the small body of research linking professional development to 
student outcomes, and to inform both the design of professional development and the preparation 
of professional developers in science. 
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Appendix B. Tables and Figures 

Not included in page count. 

Table 1 

Number of Teachers in Each Treatment Group in Each Round as of June 2009 


Site 


Round 


Teaching 

Cases 


Looking at 
Student Work 


Content 

Immersion 


Control 


Total 


1 


1 


9 


- 


8 


10 


27 




2 


- 


4 


3 


- 


7 




Follow-up 


4 


4 


5 


n/a 


13 


2 


1 


- 


7 


6 


12 


25 




2 


7 


- 


6 


- 


13 




Follow-up 


4 


5 


8 


n/a 


17 


3 


1 


11 


- 


- 


8 


19 




2 


12 


11 


- 


- 


23 




Follow-up 


6 


6 


- 


n/a 


12 


4 


1 


5 


- 


- 


6 


11 




2 


- 


7 


- 


- 


7 




Follow-up 


4 


5 


- 


n/a 


9 


5 


1 


- 


- 


9 


5 


14 




2 


- 


- 


- 


- 


- 




Follow-up 


- 


- 


- 


n/a 


- 


6 


1 


17 


- 


- 


12 


29 




2 


- 


9 


9 


- 


18 




Follow-up 


9 


4 


6 


n/a 


19 


7 


1 


- 


9 


- 


9 


18 




2 


- 


- 


6 


- 


6 




Follow-up 


- 


5 


5 


n/a 


10 


8 


1 


- 


10 


7 


17 


34 




2 


9 


8 


- 


- 


17 




Follow-up 


- 


12 


3 


n/a 


15 


Total 


1 and 2 


70 


65 


56 


79 


268 


Total 


Follow-up 


27 


41 


27 


n/a 


95 
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Table 2 

Counterbalanced Research Design with Three Treatment Models (A, B, and C) and a Co ntrol Group f 



Site 


Facilitator 

Pair 


Round 1 


Round 2 


Total no. of 
teachers at site* 


Summer 07 


Fall 07 


Winter 07/08 


Summer 08 


Site 1 


1 




c 


B 




24 




2 




A 




c 




Site 2 


3 




B 


C 




39 




4 




C 


A 






Site 3 


5 




A 


B 


A 


42 


Site 4 


6 




A 


B 




19 


Site 5 


7 




C 






17 


Site 6 


8 


A 




C 




46 




9 


A 




B 






Site 7 


10 




B 


C 




26 


Site 8 


11 




B 




A 


55 




12 




C 


B 






Totals 












268 



Note. A - Cases; B - Looking at Student Work; C - Content Immersion 

f Time distinctions are not relevant for the control group, so no D’s shown in columns, but control teachers and 
students are included in row totals. 

* Figures include only individuals with both pre- and post-instruction quiz data. 



Table 3 



Numbers of PD Courses Taught, Teacher Participants, and Students for Three Experimental 
Interventions and a Control Group in 2007-08 



Treatment 


No. times 
offered in 
Round 1 


No. times 
offered in 
Round 2 


Total no. times 
offered 


No. teachers 


No. students 


A. Teaching Cases 


5 


3 


8 


70 


1,218 


B. Looking at Student Work 


3 


5 


8 


65 


1,134 


C. Content Immersion 


4 


4 


8 


54 


1,190 


D. Control 


- 


- 


- 


79 


1,738 


Total 


12 


12 


24 


268 


5,280 



2010 SREE Conference Abstract Template 



B-2 








Table 4 

Instruments, Samples, and Data Collection Procedures 



Instrument 


Sample 


Procedure 


Written and online 
surveys 


All facilitators and teachers; follow-up 
random sample of half the teachers per 
treatment group 


Packet sent to facilitators; all treatment 
teachers take pre- and post-course, 
during first and last course sessions and 
online. ConUol teachers complete in 
project meetings at each site and 
online. 


Science content tests 


All facilitators and teachers; follow-up 
random sample of half the teachers per 
group 


Packet sent to facilitators; all treatment 
teachers take pre- and post-course, 
during first and last course sessions. 
ConUol teachers complete in project 
meetings at each site 


Student content tests 


Students of all teachers. 


Teacher administers within two weeks 
before and two weeks after elecUic 
circuits unit.* 


Teacher pedagogical 
content knowledge 
interview 


Random sub-sample of nine per group. 


Researcher administers pre and post 
interviews within two weeks before 
course, and after teacher has taught 
classroom unit. 


Classroom videotaping 
and observation 
protocol 


Two lessons in classrooms of random 
sub-sample of nine per group. 


Videographer and observer collect data 
during two lessons in elecUic circuits 
unit — first & one in middle of unit. 


Course session 
videotaping and 
observation 


All course sessions at all sites. Judith 
add detail here. 


Videographer and observer collect data 
during 3 sessions... Judith add detail 
here. 



*Sealed packets of student tests with standardized administration instructions and script sent to teachers. Teachers sign 
affidavit that they did not provide help to students other than reading test questions aloud. 



Table 5 

Teachers’ Mean Percent Correct and Gain Scores (with SDs) on Content Test by Treatment 



Treatment 


n Pre 


Pre 


n Post 


Post 


n Gain 


Gain 


A. Teaching Cases 


68 


60.9% 

(11.2) 


69 


82.2% 

(10.8) 


67 


21.9% 

(10.3) 


B. Looking at Student Work 


69 


56.3% 

(11.4) 


63 


79.3% 

(8.2) 


63 


21.5% 

(10.2) 


C. Content Immersion 


64 


57.8% 

(14.6) 


57 


80.7% 

(11.8) 


56 


22.0% 

(13.0) 


D. ConUol 


86 


56.6% 

(12.7) 


73 


59.1% 

(12.0) 


70 


2.6% 

(10.5) 
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Table 6 



Teachers’ Mean Gain Scores on Year 1 Content Pre-tests and Post-tests and Gains Maintained through 
the Follow-Up Year, by Site 





A. Teaching Cases 


B. Looking at Student Work 


C. Content Immersion 


Across-Treatments 


Site 


Y1 Gains 


Maintained 

Gains 


Y1 Gains 


Maintained 

Gains 


Y1 Gains 


Maintained 

Gains 


Y1 Gains 


Maintained 

Gains 


1 


17.4% 


8.9% 


31.3% 


34.7% 


21.1% 


27.5% 


23.3% 


23.7% 


2 


18.0% 


9.4% 


28.1% 


20.5% 


24.7% 


19.7% 


23.6% 


16.5% 


3 


25.5% 


15.9% 


13.6% 


12.4% 


- 


- 


19.6% 


14.2% 


4 


21.3% 


26.7% 


31.4% 


22.7% 


- 


- 


26.4% 


24.7% 


5 


- 


- 


- 


- 


21.1% 


13.0% 


21.1% 


13.0% 


6 


17.5% 


17.8% 


23.8% 


27.3% 


23.0% 


33.0% 


21.4% 


26.0% 


7 


- 


- 


19.9% 


22.3% 


23.8% 


25.5% 


21.8% 


23.9% 


8 


22.6% 


- 


20.4% 


14.5% 


22.1% 


12.5% 


21.7% 


13.5% 


Total 


20.4% 


15.7% 


24.1% 


22.0% 


22.6% 


21.9% 


22.4% 


19.4% 



Table 7 

HLM Results for Teacher Tests-Regressions on Teacher Gains 



Model T1 


Variable 


Coefficient 


SE 


A. Teaching Cases 


19.5 


2.3 


B. Looking at Student Work 


19.3 


2.4 


C. Content Immersion 


20.5 


2.5 


Veteran 


1.0 


1.5 


Novice 


5.2 


2.6 


Round2 


-0.3 


2.0 


Sitel 


-3.9 


3.5 


Site2 


2.5 


2.8 


Site3 


0.0 


2.8 


Site4 


2.7 


3.4 


Site5 


-3.4 


3.8 


Site7 


-2.3 


3.1 


Site8 


0.4 


2.5 


(Intercept) 


1.6 


2.4 


SD(CourselD) 


2.5 


1.5 


SD( Residual) 


10.1 


0.5 
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Table 8 

Students’ Mean Percent Correct and Gain Scores (with SDs) on Content Test by T reatment 







Pre-test 


Post-test 


Mean gain 


Treatment 


n 


mean ( SD ) 


mean (SD) 


(SD) 


A. Teaching Cases 


1347 


48 

(ID 


68 

(13) 


20 

(15) 


B. Looking at Student Work 


1392 


48 

(ID 


69 

(14) 


22 

(15) 


C. Content Immersion 


1013 


48 

(10) 


68 

(14) 


20 

(14) 


D. Control 


1494 


49 

(ID 


62 

(13) 


13 

(14) 



Table 9 

Treatment Teachers’ Students’ Mean Percent Correct and Gain Scores (with SDs) on Content 
Test by Site 



Site 


n 


Pre 


Post 


Gain 


SI 


367 


48.1 


63.5 


15.4 






(11.0) 


(13.5) 


(15.6) 


S2 


759 


48.6 


66.0 


17.4 






(10.3) 


(13.2) 


(14.2) 


S3 


1,037 


48.4 


69.3 


21.0 






(19.2) 


(12.0) 


(13.9) 


S4 


313 


48.0 


68.5 


20.5 






(11.9) 


(14.9) 


(15.8) 


S5 


366 


47.2 


61.9 


14.7 






(8.8) 


(14.2) 


(14.2) 


S6 


967 


48.7 


66.7 


17.9 






(11.0) 


(14.6) 


(15.5) 


S7 


555 


48.4 


67.3 


18.9 






(10.6) 


(13.7) 


(14.7) 


S8 


916 


47.0 


65.6 


18.6 






(11) 


(13.3) 


(15.0) 



2010 SREE Conference Abstract Template 



B-5 





Table 10 

HLM Results for Student Tests-Regressions on Student Gains 



Variable 


Coefficient 


SE 


A. Teaching Cases 


5.7 


1.4 


B. Looking at Student Work 


7.5 


1.4 


C. Content Immersion 


7.5 


1.5 


Round2 


1.6 


1.2 


Sitel 


-3.1 


1.9 


Site2 


-0.3 


1.6 


Site3 


2.9 


1.6 


Site4 


2.4 


2.1 


Site5 


-3.1 


2.3 


Site7 


1.4 


1.9 


Site8 


0.6 


1.6 


Veteran 


1.9 


0.9 


Novice 


1.0 


1.4 


(Intercept) 


11.6 


1.4 


SD(CourselD) 


1.1 


1.1 


SD(Teacher) 


5.9 


0.4 


SD( Residual) 


13.1 


0.1 



2 

First columns are coefficients; second column standard errors. (Intercept) is the grand mean, or overall intercept. 
The last three rows correspond to the standard deviations of the random intercepts due to PD course, the random 
intercept for the teachers, and the error term capturing individual student variation.. 



2010 SREE Conference Abstract Template 



B 




